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FOREWORD 


This  report  is  an  outgrowth  of  the  work  of  the 
Defense  Science  Board  Task  Force  on  Test  and  Evaluation, 
and  the  checklists  herein  have  been  derived  from  the 
study  of  past  major  weapon  system  programs. 

The  T&E  expert  in  reading  this  volume  will  find 
many  precepts  which  will  strike  him  as  being  too  obvious 
to  be  included  in  checklists  of  this  type.  These  items 
are  included  because  examples  were  found  where  even  the 
obvious  has  been  neglected,  not  because  of  incompetence 
or  lack  of  personal  dedication  by  the  people  i  i  charge 
of  the  program,  but  because  of  financial  and  temporal 
pressures  which  forced  competent  managers  to  compromise 
on  their  principles.  It  is  hoped  that  the  inclusion  of 
the  obvious  will  prevent  repetition  of  the  serious  errors 
which  have  been  made  in  the  past  when  such  political, 
economic  and  temporal  pressures  have  forced  project 
managers  to  depart  from  the  rules  of  sound  engineering 
practices. 

In  the  long  run,  taking  short  cuts  during  T&E  to 
save  time  and  money  will  result  in  significant  increases 
in  the  overall  costs  of  the  programs  and  in  the  delay  of 
the  delivery  of  the  corresponding  weapon  systems  to  the 
combatant  forces. 
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T&E  GUIDELINES  FOR  SHIP  SYSTEMS 


The  checklist  items  presented  here  are  specifically  applicable  to 
ship  testing  and  evaluation.  It  is  suggested  that  the  user  of  this  volume 
also  refer  to  the  Report  of  the  Defense  Science  Board  on  Test  and  Evaluation 
which  contains  general  checklist  iters  also  applicable  to  this  system  T&E 
program.  The  checklist  items  presented  here  are  organized  into  time  phases 
of  the  acquisition  process  oriented  to  the  DSARC  cycle. 

The  checklists  cover  various  aspects  of  the  major  activities  that 
should  be  underway  during  a  given  time  period.  Hence,  a  checklist  might 
cover  the  (1)  evaluation  of  work  that  occurred  in  the  previous  phase, 

(2)  conduct  of  tests  planned  in  the  previous  phase  and  executed  in  the 
subject  phase,  and  (3)  plans  and  other  preparatory  actions  for  test  sche¬ 
dules  to  be  conducted  in  a  subsequent  phase.  For  reasons  such  as  this, 
items  on  some  subjects,  such  as  development  test  plans,  may  appear  in  more 
than  one  phase.  In  addition,  since  the  Services  and  the  DSARC  have  flexi¬ 
bility  in  deciding  how  rapidly  to  progress  in  the  validation  phase,  there 
may  be  cases  where  the  Request  for  Proposals  (RFPs),  proposal  evaluations, 
source  selections,  or  contract  negotiations  may  occur  after  the  DSARC 
approves  full-scale  development  instead  of  before.  For  this  reason,  it 
is  recommended  that  previous  checklists  in  the  Validation  Phase  be  re¬ 
viewed  when  entering  the  Full-Scale  Engineering  Development  Phase.  The 
following  are  the  phases  used  in  this  report. 

CONCEPTUAL  PHASE 

The  checklist  items  in  this  phase  are  for  guidance  in  evaluating  T&E 
activities  during  the  Conceptual  Phase  of  weapon  systems  acquisition. 

This  phase  (often  research  and  exploratory  development)  precedes  the  first 
DSARC  milestone  and  is  focused  on  the  development  of  a  weapon  system  con¬ 
cept  that  offers  high  prospects  of  satisfying  an  identified  military  need. 

Although  not  called  for  in  DoD  Directive  5000.1  specifically,  the 
objectives  of  this  phase  should  be: 
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1.  To  verify  that  there  Is  a  military  need  for  the  proposed 
system. 

2.  To  demonstrate  that  there  is  a  sound  physical  basis  for  a  new 
weapon  system. 

3.  To  formulate  a  concept,  based  on  demonstrated  physical 
phenomena,  for  satisfying  the  military  need. 

4.  To  show  that  the  proposed  solution  is  superior  to  its  com¬ 
petitors  in  terms  of  potential  effectiveness,  probability  of 
success,  probable  cost,  impact  on  the  U.S.  military  posture, 
and  development  risks. 

5.  To  analyze  the  technology  outlook  and  the  military  need  to 
show  that  it  is  better  to  start  advanced  development  now 
rather  than  to  wait  for  future  technological  improvements. 

6.  To  Identify  the  key  risk  areas  and  critical  issues  that  need 
to  be  resolved  before  full-scale  development  is  initiated. 

The  most  Important  product  of  this  phase  is  the  Development  Concept 
Paper  (DCP)  or  its  equivalent.  The  DCP  defines  program  issues,  including 
special  logistics  problems,  program  objectives,  program  plans,  performance 
parameters,  areas  of  major  risk,  system  alternatives,  and  acquisition 
strategy. 


VALIDATION  PHASE 

The  checklist  items  in  this  phase  are  for  guidance  in  conducting  T&E 
during  the  Validation  Phase  (the  time  between  when  the  DSARC  recommends 
approval  of  the  DCP  for  the  first  time  and  when  the  DSARC  recommends  full- 
scale  development  of  the  system) . 

'While  these  objectives  are  not  spelled  out  in  the  DoD  Directive  5000.1, 
the  objectives  of  the  Validation  Phase  should  be  to  confirm: 

1.  The  need  for  the  selected  system  in  consideration  of  the  threat, 
system  alternatives,  special  logistics  needs,  estimates  of 
development  costs,  preliminary  estimates  of  life  cycle  costs 
and  potential  benefits  in  context  with  overall  DoD  strategy  and 
fiscal  guidance. 

2.  The  validity  of  the  operational  concept. 

3.  That  development  risks  have  been  identified  end  solutions  are 
in  hand. 

4.  Realism  of  the  plan  for  full-scale  development. 
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In  the  pursuit  of  the  above  objectives,  it  is  likely  that  advanced 
development  T&E  will  be  conducted  to  resolve  issues.  In  some  cases,  an 
RFP  for  full-scale  engineering  development  will  be  prepared,  proposals 
will  be  received  and  evaluated,  and  contracts  negotiated  in  preparation 
for  seeking  DSARC  approval  for  the  next  phase.  Therefore,  some  checklist 
items  are  included  to  help  ensure  that  this  work  properly  reflects  the 
T&E  interests  in  this  and  subsequent  phases.  For  example,  the  RFP  must 
Include  adequate  guidance  to  ensure  that  sufficient  resources  and  time  are 
available  so  that  engineering  effort  can  properly  support  the  initial  DT&E 
with  hardware,  software,  technical  data,  and  training. 

The  primary  emphasis  of  OSD/T&E  activities  is  with  items  3  and  4  above. 
Special  attention  should  be  given  to  the  planning  of  IOT&E  activity  as  it 
is  Incorporated  in  the  engineering  development  contract  as  well  as  the 
DT&E  associated  with  addressing  the  critical  issues  and  areas  of  major 
risk  identified  in  the  DCP. 

FULL-SCALE  ENGINEERING  DEVELOPMENT  PHASE 

The  checklist  items  contained  in  this  phase  are  for  guidance  in 
conducting  T&E  during  the  Full-Scale  Engineering  Development  Phase.  This 
Includes  the  major  DT&E  and  the  IOT&E  conducted  prior  to  the  major  pro¬ 
duction  decision.  By  this  time,  the  weapon  system  is  well-defined  and  is 
becoming  a  unique  item  and,  hence,  sound  judgment  must  be  applied  in  using 
these  checklist  items. 

To  enter  the  Engineering  Development  Phase,  the  DSARC  will  have: 

•  Confirmed  the  need  in  consideration  of  the  threat,  alternatives, 
logistic  needs,  cost,  and  benefits. 

•  Identified  development  risks. 

•  Confirmed  the  realism  of  the  development  plan. 

Given  the  above,  the  primary  objectives  of  the  DT&E  should  be  to: 

1.  Demonstrate  that  the  engineering  and  design  and  development 
process  is  complete  and  that  the  design  risk';  have  been  mini¬ 
mized  (the  system  is  ready  for  production) . 

2.  Demonstrate  that  the  system  will  meet  specifications. 
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The  primary  objectives  of  the  IOT&E  should  be  to: 

3.  Assess  operational  suitability  and  effectiveness. 

4.  Validate  organizational  and  employment  concepts. 

5.  Determine  training  and  logistic  requirements. 

In  addition,  the  validity  of  the  plan  for  the  remainder  of  the  program 
must  be  confirmed  by  the  DSARC  before  substantial  product  ion/ deployment 
will  be  recommended  to  the  Secretary  of  Defense. 

The  level  of  OSD/T&E  activity  is  highest  during  this  phase.  The 
IOT&E  plan  must  be  designed,  the  tests  conducted,  and  the  data  analyzed 
to  evaluate  the  inputs  associated  with  the  primary  objectives.  These 
tests  should  not  be  conducted  until  the  primary  objectives  of  the  DT&E 
have  been  met.  Thus,  OSD/T&E  activity  is  required  to  assess  that  the  DT&E 
major  milestone — the  system  is  ready  for  production— has  been  achieved. 
Close  monitoring  of  the  T&E  Service  activity  is  required  during  the  latter 
stages  of  this  phase. 

SUBSTANTIAL  PRODUCTION/DEPLOYMENT  PHASE 

The  checklist  items  contained  in  this  phase  are  for  guidance  in  con¬ 
ducting  T&E  after  the  substantial  production  decision  has  been  made  by  the 
DSARC.  This  includes  DT&E  and  follow-on  OT&E  to  be  conducted  on  the 
early  production  items. 

To  enter  the  Production/Deployiqent  Phase,  the  DSARC  will  have  re¬ 
viewed  the  program  to  confirm: 

•  The  need  for  the  system. 

•  A  practical  engineering  design  with  adequate  consideration  of 
production  and  logistic  problems  is  complete. 

•  All  technical  uncertainties  have  been  resolved  and  opera¬ 
tional  suitability  has  been  determined  by  T&E. 

•  The  realism  of  the  plan. 

The  primary  objective  of  the  DT&E  in  this  phase  should  be  to: 

1.  Verify  that  the  production  system  meets  specifications. 

The  primary  objectives  of  the  follow-on  OT&E  should  be  to: 
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2.  Validate  the  operational  suitability  and  effectiveness. 

3.  Optimize  organization  and  doctrine. 

4.  Validate  training  and  logistic  requirements. 

At  this  point,  the  OSD/T&E  activity  is  similar  to  that  in  the 
previous  phase;  however,  much  of  the  testing  is  verification  that  the 
production  system  performance  is  as  expected.  Hence,  most  of  the  items 
in  the  previous  phase  are  appropriate  to  this  phase,  especially  tho3e 
related  to  OT&E. 


I.  CONCEPTUAL  PHASE 


?  HOT 


During  the  Conceptual  Phase  of  an  acquisition  program  for  a  new  ship 
class,  the  primary  efforts  are  normally  devoted  to  establishing  the  need 
for  a  program  to  meet  a  particular  operational  requirement  in  the  face  of 
the  projected  threat.  Initial  plans  are  made  regarding  schedules,  costs, 
risks  and  issues.  Included  in  this  phase  are  schedules  of  T&E  milestones 
and  such  critical  issues  as  are  known  in  this  early  phase  of  the  program. 
Where  combat  system  (land  based  or  sea  based)  testing  is  to  be  a  part  of 
the  program,  early  configuration  planning  is  essential.  Feasibility 
testing  of  new  hull  or  propulsion  concepts,  sufficient  to  justify  proce¬ 
eding  with  the  program  should  be  conducted. 

These  and  similar  matters  are  usually  formulated  in  the  initial  DCP 
and  presented  to  the  DSARC  as  a  basis  for  establishment  of  the  ship 
acquisition  program. 

Conceptual  Phase  checklist  items  are  as  follows: 

1.  Test  and  Evaluation  Plan 

2.  Test  Objectives  and  Critical  Issues 

3.  OT&E  Phasing 

4.  Test  Facilities  and  Instrumentation  Requirements 

5.  Multiple  Approach  to  Weapon  System  Development 

6.  Comparison  of  New  Versus  Old  System 

7.  Test  Support  Facilities 

8.  Fleet  Operating  Force  Requirements 

9.  Mission-Related  Measures  of  Effectiveness 

10.  Ship  T&E  Management 

11.  T&E  of  Large,  Integrally-Constructed  Systems 


Preceding  page  blank 
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TEST  AND  EVALUATION  PLAN 


Prior  to  DSARC  I,  sufficient  material  should  be  generated  to  allow 

for  an  evaluation  of  the  overall  T&E  program. 

As  part  of  this,  a  test  and  evaluation  concept  for  those  tests  to  be 
conducted  prior  to  DSARC  II  (to  validate  the  concept  and  hardware  approach 
to  the  weapon  system)  should  also  be  developed.  The  plan  must  include 
statements  of: 

(a)  The  critical  issues  and  the  overall  purpose  of  the  test, 
such  as  service  approval  of  weapon  subsystem  prior  to 
commencement  of  integrated  testing. 

(b)  The  operational  requirements,  such  as  desired  reliability 
and  maintainability  goals. 

(c)  The  major  test  objectives. 

(d)  The  schedule  of  test  milestones. 

(e)  The  major  resources  required. 

•  Test  environment,  facilities  and  instrumentation 

•  Simulated  operational  environment 

•  Completed  maintenance  documentation. 

(f)  The  organizations  which  will  conduct  the  test  program. 

(g)  The  analysis  and  evaluation  approach. 

2.  TEST  OBJECTIVES  AND  CRITICAL  ISSUES 

In  evaluatina,  the  initial  test  concept,  it  is  important  that  the 

test  objectives  during  the  time  period  from  DSARC  I  to  DSARC  II 

address  the  major  critical  issues,  especially  technological  issues. 

As  an  example,  if  a  shipboard  weapon  system  contemplates  use  of  a 
new  method  of  target  detection  or  acquisition,  this  should  be  evaluated 
as  to  feasibility  by  test  during  advanced  development.  Test  objectives 
should  be  simply  stated  and  given  priorities.  A  plan  for  the  conduct 
of  the  test  and  the  data  collection,  reduction,  and  analysis  must  be  in 
sufficient  detail  so  that  one  can  readily  evaluate  the  performance  of 
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Che  system  and  whether  or  not  the  test  objectives  can  be  met.  The 
relationship  between  the  identified  performance  parameters  and  the  ex¬ 
pected  test  results  should  be  established  prior  to  the  conduct  of  the  test. 
Further,  the  set  of  objectives  for  each  of  the  tests  should  be  clearly 
related  to  the  defined  program  objective.  When  this  relationship  is  not 
clear,  amplifying  data  should  be  required.  The  following  should  be 
addressed. 

•  Issues  associated  with  system  integration  (e.g.,  placement 
of  radar/radio  antennas  to  avoid  RFI  problems) . 

•  Issues  associated  with  location  of  major  subsystems  on  the 
ship  (e.g.,  sonar  domes,  gun  mounts,  weapon  launchers). 

•  Issues  associated  with  combat  systems. 

•  Issues  associated  with  propulsion  systems. 


3.  OT&E  PHASING 

In  evaluating  test  plans,  look  favorably  on  phasing  where  the  OT&E 

is  run  in  parallel  with  continued  DT&E. 

Problems  that  become  apparent  in  the  operational  testing  can  often  be 
evaluated  much  more  quickly  and  more  completely  with  the  instrumented  DT&E 
hardware.  This  Is  most  attractive  where  the  DT&E  is  performed  with  non¬ 
expendable  hardware,  and  is  particularly  applicable  to  ship  systems  where 
the  same  test  installation  and  crew  members  are  used  to  perform  DT&E  and 
OT&E. 

In  general,  test  plans  should  make  provisions  for  the  occurrence  of 
failures,  should  include  time  and  money  necessary  for  investigating  test 
failures  and  make  provisions  for  elimination  of  the  cause  before  another 
test  series.  Experience  has  shown  this  to  be  particularly  true  in  software 
integration  testing.  As  an  example,  ship  C&C  integration  test  sites  have 
frequently  been  unable  to  maintain  schedule,  simply  because  this  factor  was 
not  recognized.  A  percentage  of  the  total  tests  (sorties,  runs,  trials, 
experiments)  should  be  allocated  to  retesting,  over  and  above  the  number 
need  if  all  tests  are  successful. 
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4.  TEST  FACILITIES  AND  INSTRUMENTATION  REQUIREMENTS 

Before  DSARC  I  the  test  facilities  and  instrumentation  requirements 
to  conduct  developmental  and  operational  tests  should  be  Identified, 
a1ong  with  a  tentative  schedule  of  test  activities. 

The  applicability  of  the  test  ranges  and  the  adequacy  of  the  facilities 
and  instrumentation  should  be  verified.  Insofar  as  possible  alternative 
approaches  (different  ranges,  etc.)  and  instrumentation  Improvements  needed 
should  be  specified.  Of  prime  importance  are  the  constraints  to  be  placed 
on  the  test  because  of  the  range  and  instrumentation.  If  range  and  instrumen¬ 
tation  factors  are  found  to  cast  significant  doubt  on  the  meaningfulness  of 
the  test  data,  the  steps  necessary  to  assure  meaningful  data  should  be  identi¬ 
fied  and  planned  before  inclusion  in  the  test  concept. 

Where  applicable,  instrumentation  necessary  for  T&E  of  the  new  ship,  such 
as  strain  gauges,  torque  indicators,  and  position  repeaters,  should  ,  for  economy, 
be  made  part  of  the. lead  ship  design  and  built  into  it,  rather  than  added  later 
as  an  afterthought. 

5.  MULTIPLE  APPROACH  TO  WEAPON  SYSTEM  DEVELOPMENT 

Whenever  possible,  the  weapon  system  concept  should  not  be  predicated 
on  the  successful  development  of  a  single  hardware  or  software  approach 
in  the  various  critical  subsystems  (unless  it  has  been  previously  de¬ 
monstrated  adequately) . 

Instead,  each  should  have  several  potential  ways  of  being  achieved. 

When  multiple  approaches  are  possible,  critical  advanced  development 
activities  should  proceed  with  at  least  two  approaches,  with  T&E  being  planned 
for  the  breadboard  hardware  resulting  from  each  approach.  As  an  example, 
in  the  case  of  some  new  ship  classes,  multiple  prototypes  have  been  planned 
in  order  to  more  quickly  resolve  the  uncertainties  in  the  approaches  to  be 
taken  for  weapon  system  and  type  of  propulsion.  Tests  to  compare  the  performance 
of  the  breadboard  items  should  also  be  planned.  For  subsystems  for  which  a 
single  approach  is  used,  tests  should  be  planned  which  evaluate  the  breadboard 


under  many  degraded  modes  to  Investigate  the  potential  limitations  of 
that  approach. 

6.  COMPARISON  OF  NEW  VERSUS  OLD  SYSTEM 

The  procedure  for  examining  the  relative  performance  of  new  or  modi¬ 
fied  systems  versus  old  should  be  indicated  in  the'  T&E  plan. 

In  the  interests  of  updating  the  technology  of  systems  there  has 
been  a  tendency  to  ignore  or  minimize  the  operational  value  of  proven 
systems.  Sometimes  new  or  modified  systems  have  yielded  relatively  minor 
improvements  in  effectiveness  at  considerable  additional  cost  when  com¬ 
pared  with  the  proven  system.  For  example,  a  sonar  system  Intended  to 
provide  improved  detection  capability  was  found  not  to  be  significantly 
better  than  its  predecessor.  Unfortunately,  this  was  not  discovered 
until  after  a  large  number  had  been  procured. 

7.  TEST  SUPPORT  FACILITIES 

The  phasing  of  test  support  facilities  must  be  carefully  planned. 
with  some  schedule  flexibility  to  cover  late  deliveries  and  other 
unforeseen  problems. 

Some  systems  can't  be  fully  tested  without  shore  test  facilities. 

The  long  lead  times  to  obtain  authorization,  appropriations,  and  to  con¬ 
struct  facilities  can  pace  a  program.  For  example,  many  steps  and  con¬ 
siderable  time  will  be  involved  in  constructing  a  land  based  combat  systems 
test  site  with  test  instrumentation  and  equipment  checked  out  and  ready 
to  support  tests. 

8.  FLEET  OPERATING  FORCE  REQUIREMENTS 

The  requirement  for  fleet  operating  forces  for  DT&E  or  OT&E  should 
be  assessed  early  in  the  program,  and  a  specific  commitment  made 
as  to  the  types  of  units  to  be  employed. 

As  the  program  progresses,  the  necessary  fleet  support  should  be 
formally  included  in  the  fleet  employment  schedules.  To  this  end 
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COMOPTGVFOR  assistance  in  the  test  planning  should  begin  early.  Since 
operational  forces  for  test  purposes  are  always  in  short  supply,  vital 
tests  may  not  be  performed  in  time  to  have  an  effect  on  the  program  if 
these  actions  are  not  taken. 

9.  MISSION-RELATED  MEASURES  OF  EFFECTIVENESS 

During  the  Conceptual  Phase  of  the  acquisition  of  a  new  class  of 
ship,  a  study  effort  should  be  commenced  jointly  by  the  CNO  and 
COMOPTEVFOR  to  establish  mission-related  measures  of  effectiveness 
which  may  be  expressed  in  numerical  fashion  and  which  may  later  be 
made  the  subject  of  OT&E  to  determine  how  closely  the  new  ship 
system  meets  the  operational  need  for  which  it  was  conceived. 

As  an  example,  for  a  surface  ship  or  submarine  whose  primary  mission 
is  to  be  ASW:  threat-related  parameters  can  be  established  based  on  known 
adversary  capabilities;  own  ship  parameters  can  be  set  according  to  the 
capabilities  of  present  equipment  or  those  demanded  of  equipment  under 
development;  and  environmental  factors  can  be  entered  using  results  of 
past  and  present  investigations.  Based  on  such  study,  a  numerical  prob¬ 
ability  of  success  can  be  generated  in  each  of  the  mission  areas  con¬ 
ceived  for  the  ship,  after  which  acceptable  minimum  standards  can  be 
established  as  test  criteria  for  operational  suitability.  Advantages  of 
this  process  are  that  it  will  serve  to  eliminate  bias  and  subjectivity 
from  the  analysis  of  test  results  thereby  increasing  their  credibility, 
and  will  allow  greater  use  of  computer  simulation  and  modeling  of  the 
operational  scenario  with  attendant  reduction  in  the  amount  of  live  test¬ 
ing. 


10.  SHIP  T&E  MANAGEMENT 

The  management  of  ship  T&E  should  ensure  that  test  requirements  are 
necessary  and  consistent  relative  to  system/subsystem  aspects  and 
that  the  necessary  testing  is  coordinated  so  that  test  redundancy 
does  not  become  a  problem. 


Ship  systems  are  large,  complex  and  comprised  of  many  subsystems, 
some  critical  to  the  primary  mission  of  the  ship  and  all  important  to  the 
functioning  of  ship  activities. 
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11.  T&E  OF  LARGE,  INTEGRALLY-CONSTRUCTED  SYSTEMS 

Major  subsystems  should  be  proven  feasible  prior  to  firm  commit¬ 
ment  to  a  detail  hull  design. 

In  any  situation  where  a  subsystem  must  be  Integrally  constructed  with 
the  ship  (i.e.,  with  the  ship  essentially  built  around  the  subsystem)  such 
as  large,  high-power  sonars,  phased  array  radars  or  complex  missile 
launching  systems,  T&E  planning  should  be  such  that  sufficient  testing 
of  the  subsystem  will  be  completed  early  enough  to  provide  confidence 
in  the  continued  development  and  probable  ultimate  performance  and  opera¬ 
tional  suitability  prior  to  firm  commitment  to  a  detailed  hull  design. 
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II.  VALIDATION  PHASE 

Although  ship  acquisition  programs  are  of  varying  complexity  and  each 
is  somewhat  different  in  time  phasing  of  this  part  of  the  cycle,  it  is 
possible  to  state  generally  some  usual  characteristics  of  the  Validation 
Phase.  This  may  involve  the  execution  of  one  or  more  preliminary  design 
contracts  leading  to  proposals  from  which  later  a  single  configuration  is 
to  be  selected.  DT&E  and  possibly  some  IOT&E  on  expected  major  subsystems 
may  be  proceeding.  Development  milestones,  including  DT&E,  will  be  plan¬ 
ned  in  considerable  detail.  Where  radically  new  designs  in  hull  or  pro¬ 
pulsion  are  Involved,  test  prototypes  may  be  constructed  and  testing  begun. 
If  COMOPTEVFOR  participation  in  test  planning  was  not  begun  during  the 
Conceptual  Phase,  it  must  begin  here,  both  with  respect  to  individual  sub¬ 
systems  (weapons,  sensors,  propulsion,  etc.)  and  to  integrated  system 
testing.  Cost  and  schedule  of  the  acquisition  are  given  additional  firm¬ 
ness.  During  this  phase  sufficient  information  should  be  accumulated 
upon  which  to  base  a  decision  regarding  embarkation  on  full-scale  develop¬ 
ment. 

Validation  Phase  checklist  items  are  as  follows: 

1.  COMOPTEVFOR  Participation  in  DT&E 

2.  Authentication  of  Human  Factors  Concepts 

3.  Government  Commitments 

4.  Contract  Form  and  Incentives 

5.  Acquisition  Strategy 

6.  Performance  History  of  Components 

7.  Effect  of  Supposedly  Minor  Modifications 

8.  End-to-End  Study 

9.  Ocean  Parameters  for  ASW  Systems 

10.  Criteria  for  Selection  Among  Competitive  Designs 

11.  Evaluation  of  Results  of  Exploratory  Testing 

12.  Prototyping  for  Technical  Risk 

13.  Prototyping  of  Competing  Systems 

14.  T&E  of  Command/Operating  Centers 

Preceding  page  blank  is 
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15.  Software  Testing 

16.  New  Hull  Forms 

17.  Effect  of  Hull  and  Propulsion  on  Mission  Capabilities 

18.  Advances  in  Propulsion 

19.  Propulsion  System  in  Use  in  Other  Classes 

20.  IOT&E  Shipboard  Gun  Systems 

21.  Targets  for  AAW  OT&E 

22.  Waivers  to  T&E  of  Ship  Systems 

23.  Environmental  Effects  on  Sonar  Domes 

24.  Hull/Machinery  Testing  by  Computer  Simulation 

25.  Operational  Reliability 

26.  Inter-shipyard  Coordination 

The  reader  should  also  refer  back  to  the  checklist  items  in  the  Con¬ 
ceptual  Phase  since  many  of  these  items  will  also  be  applicable  to  the 
Validation  Phase. 
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1.  COMOPTEVFOR  PARTICIPATION  IN  DT&E 

It  Is  imperative  that  COMOPTEVFOR  participate  In  all  of  the  T&E 
phases  to  ensure  that  the  user  needs  are  represented  in  the  develop¬ 
ment  of  the  system  concept  and  hardware. 

Initially,  COMOPTEVFOR  should  play  an  advisor  role  during  the  feasi¬ 
bility  and  engineering  testing,  and  gradually  assume  more  leadership  in 
the  conduct  of  the  testing  program  as  it  becomes  more  and  more  operational. 
This  should  facilitate  the  necessary  communication  and  interaction  between 
the  developing  and  user  command — especially  needed  during  the  DT&E  and 
IOT&E  phases. 


2.  AUTHENTICATION  OF  HUMAN  FACTORS  CONCEPTS 

At  an  appropriate  time  in  concept  definition  or  development  phase, 
T&E  should  authenticate  the  human  factors  concepts  embodied  in  the 
proposed  system  design,  examining  questions  of  safety,  comfort, 
appropriateness  of  man-machine  interfaces,  as  well  as  the  number 
and  skill  levels  of  the  personnel  required. 


The  numbers  of  personnel  required  should  be  validated  against  both 
operational  and  maintenance  requirements.  Testing  early  versions  In  the 
"human  acceptability  and  compatibility"  environment  is  extrdmely  important. 
This  will  also  help  to  validate  the  manning  requirements. 


3.  GOVERNMENT  COMMITMENTS 

GFE  items  should  be  accounted  for  in  the  T&E  program 


If  there  are  GFE  and  other  government  commitments  in  the  proposed 
contract,  be  concerned  about  the  following: 

(a)  Can  the  gear  with  required  performance  be  available  when 
required? 

(b)  Can  government  supported  facilities  provide  the  assistance 
required  at  the  time  needed?  If  not,  is  it  reasonable  to 
construct  the  required  facilities  (test  range,  instrumentation, 
building,  etc.)?  If  not,  what  alternatives  are  available? 
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(c)  Avoid  contract  terms  that  vaguely  commit  the  government. 
Don't  include,  "government  support  as  required"  or  "test 
facilities  will  be  made  available  when  needed". 


4.  CONTRACT  FORM  AND  INCENTIVES 

Ensure  that  development  test  and  evaluation  problems  are  not  inten¬ 
sified  by  the  contract  form  or  factors  closely  tied  to  the  contract. 

To  minimize  these  problems,  considerable  attention  is  required  in  the 
contract  wording.  Improper  incentives  can  warp  the  proper  conduct  of  the 
test  and  evaluation.  In  designing  contractually  required  demonstration 
tests,  upon  whose  outcome  may  depend  large  incentive  payments,  or  even 
program  continuation,  it  is  essential  to  specify  broader  success  criteria 
than  simply  a  ver  few  points  in  the  performance  envelope.  If  this  is  not 
done,  the  entire  program  may  be  skewed  to  meet  the  requirements  of  the 
selected  scenario,  to  the  detriment  of  exploring  the  entire  performance 
envelope.  For  example,  a  large  incentive  attached  to  maximum  sustained 
speed  might  be  self-defeating  if  habitability  and  survivability  during 
high  sea  states  are  not  also  designed  into  the  new  ship. 

The  contract  should  be  designed,  if  possible,  so  that  necessary 
changes  can  be  quickly  made  as  the  development  unfolds  and  appropriate 
testing  activity  introduced. 

5.  ACQUISITION  STRATEGY 

The  acquisition  strategy  for  a  ship  and  its  subsystems  should  allow 
for  a  sufficient  time  between  the  planned  end  of  demonstration  test¬ 
ing  and  major  procurement  decisions  of  government  furnished  equip¬ 
ment  so  that  there  is  a  flexibility  for  modification  of  plans  which 
may  be  required  during  the  test  phases  of  the  program. 

The  shipbuilding  schedule  will  require  the  delivery  of  GFE  to  the 
shipbuilder  at  prescribed  times.  If  the  test  program  for  GFE  is  not 
planned  carefully,  the  shipbuilding  schedule  may  force  procurement  deci¬ 
sions  on  GFE  prior  to  completion  of  testing. 

The  acquisition  strategy  for  the  system  should: 
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(a)  Ensure  that  sufficient  dollars  are  available  not  only  to  con¬ 
duct  the  planned  T&E  but  to  allow  for  the  additional  T&E 
which  is  always  required  due  to  failures,  design  changes,  etc. 

(b)  Be  evaluated  to  minimize  the  so-called  T&E  gap  caused  by  a 
lack  of  hardware.  Specifically,  a  test  gap  can  result  if 
funds  are  not  applied  until  the  results  of  IOT&E  are  known 
because  of  the  required  lead  time  for  production  planning, 
production  facilities  and  tool  and  production  hardware. 


6.  PERFORMANCE  HISTORY  OF  COMPONENTS 

Performance  histories  on  non-expendable  weapon  systems  should  be 
maintained. 


When  developing,  testing  and  evaluating  the  various  subsystems  (and 
systems)  of  non-expendable  weapon  systems,  each  component  of  the  systems 
should  be  numbered  and  a  performance  history  kept  which  allows  an  analysis 
of  that  component's  performance,  with  respect  to  reliability,  maintain¬ 
ability  and  availability. 


7.  EFFECT  OF  SUPPOSEDLY  MINOR  MODIFICATIONS 

It  is  important  to  identify  what  changes  are  contemplated,  even  in 
the  proven  systems.  Major,  and  sometimes  minor,  modifications  to 
systems  can  sometimes  be  as  troublesome  and  costly  as  new  system 
development. 


On  a  number  of  occasions  in  the  past,  minor  modifications  changed 
the  basic  system's  physical  and  operational  characteristics  enough  to 
cause  major  interface  problems  when  consolidated  with  other  systems. 

As  an  example,  certain  shipboard  radars  underwent  OT&E  and  were  accepted 
for  service  use  in  the  mid-1960s,  but  were  not  produced  in  quantity  at 
that  time.  Recent  modifications  which  are  designed  to  bring  these 
equipments  to  1970 's  state-of-the-art  and  production  technology  should 
not  necessarily  be  treated  as  minor  modifications  and  may  require  fur¬ 
ther  testing  to  assure  operational  suitability. 

8.  END-TO-END  STUDY 

A  comp lex  system  needs  "end-to-end"  study. 
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Whenever  two  major  systems  are  required  to  be  operated  in  conjunction 
(e.g.,  a  radar,  a  C&C  system,  a  fire  control  system  and  indeed  the  inter¬ 
action  of  a  subsystem  with  the  ship  itself),  test  and  evaluation  plans 
should  provide  for  tests  of  the  interface (s)  and  should  monitor  changes 
therein.  For  example,  the  wake  and  rooster  tail  characteristics  of  a  ship 
are  crucial  to  the  streaming,  towing,  and  recovery  characteristics  of 
a  towed  sonar  system.  T&E  of  the  hoist  and  towed  sonar  system  should  be 
performed  on  the  class  on  which  the  sonar  system  will  be  installed  or  on 
a  ship  that  has  exactly  the  same  wake  and  rooster  tail  characteristics. 

Example:  the  USS  GLOVER  (AGDE-1)  was  not  a  good  test  ship  for  the  DE  1052 
VDS  for  checking  hoist  and  towed  body  behavior  even  though  GLOVER  had 
many  characteristics  similar  to  those  of  the  DE  1052.  Furthermore,  it 
was  not  anticipated  that  major  water  intrusion  would  occur  in  the  open 
hoist  room  of  the  DE  1052  class — but  this  turned  out  to  be  a  major  problem. 

The  many  subsystems  which  are  incorporated  in  ships  dictate  that  Inter¬ 
face  testing  be  conducted  to  define  inputs  for  configuration  control,  com¬ 
puter  capacity,  power  and  cooling  capacity,  electrical  capability,  etc. 

The  totality  of  the  systems  definition  based  on  realistic  tests  which 
simulate  the  operational  environment  to  the  maximum  extent  possible  must 
be  clearly  understood  and  refined  early,  particularly  in  acquisition  of 
our  modern  complex  ships. 

9.  OCEAN  PARAMETERS  FOR  ASW  SYSTEMS 

Shipboard  ASW  combat  systems  must  be  designed  to  operate  effectively 
under  sea  conditions  encountered  in  all  seasons  in  all  the  .world’s 
oceans. 

It  is  unreasonable,  however,  to  conduct  tests  everywhere  to  validate 
this  capability.  Alternatively,  many  studies  which  have  been  conducted  in 
the  past  and  are  still  on-going  have  produced  a  sufficient  body  of  data  on 
many  of  the  ocean  areas  of  the  world  to  allow  accurate  prediction  of  parameters 
important  to  undersea  warfare.  These  can  and  should  be  used  early  in  the 
program  to  develop  models  for  computer  simulations  to  estimate  the  effectiveness 


20 


of  systems  contemplated  for  ships.  Adequate  at-sea  testing  should  be  conducted 
at  such  point  in  the  program  as  practicable  to  validate  the  simulations.  The 
technique  is  particularly  useful  in  comparing  alternative  suites  proposed  for 
a  new  class  of  ship,  although  final  evaluation  must  await  the  availability  of 
actual  hardware. 


10.  CRITERIA  FOR  SELECTION  AMONG  COMPETITIVE  DESIGNS 

Whenever  competitive  designs  are  under  consideration,  criteria  for 
selection  should  be  specified  in  advance,  with  critical  issues 
identified  for  each  design. 

Planning  should  include  the  evaluation  criteria  to  be  used  for  the  selec¬ 
tion  of  the  final  system  design.  They  should  be  based  largely  on  performance 
factors  which  are  measurable  through  testing.  A  data  collection  and  evalua¬ 
tion  plan  should  be  developed  which  describes  the  range  of  acceptable  per¬ 
formance  for  each  factor.  This  applies  particularly  to  systems  for 
installation  aboard  ship.  For  example,  the  many  trade-offs  between  cost, 
performance  and  physical  size  associated  with  shipboard  weapon  systems 
planned  for  quantity  procurement  dictate  the  need  for  hard  evaluation 
criteria.  Those  criteria  which  are  capable  of  test  verification  should 
be  addressed  in  early  test  planning 


11.  EVALUATION  OF  RESULTS  OF  EXPLORATORY  TESTING 

Results  of  tests  conducted  during  exploratory  development  and  which 
most  likely  have  been  conducted  on  brassboard,  breadboard,  or  modi¬ 
fied  existing  hardware  should  be  carefully  evaluated. 


Special  attention  should  be  given  to  items  such  as: 

(a)  The  packaging  of  the  hardware  may  significantly  affect  the 
performance  characteristics  so  that  the  suggested  proof  of 
validation  is  inconclusive,  e.g.,  mock-ups  of  operating  spaces 
should  be  examined  to  determine  the  degree  to  which  they  dup¬ 
licate  the  final  installation,  differences  should  be  evaluated. 
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(b)  The  laboratory-type  environment  in  which  the  hardware  was 
tested  may  preclude  the  generation  of  data  needed  to  vali¬ 
date  that  the  concept  and  technology  approach  will  be  appli¬ 
cable  to  an  operational  environment. 

12.  PROTOTYPING  FOR  TECHNICAL  RISK 

When  high  technical  risk  is  present,  development  should  be  structured 
around  the  use  of  prototypes  designed  to  prove  the  system  concept 
under  realistic  operational  conditions  before  proceeding  to  engineer¬ 
ing  development. 


It  is  good  to  take  a  risk;  however*  when  an  implied  commitment  to  pro¬ 
duction  is  involved  the  technology  should  be  operationally  proof  tested  prior 
to  commencing  full-scale  development.  On  the  other  hand,  avoid  the  tempta¬ 
tion  of  thinking  that  anything  is  "state-of-the-art"  until  it  is  working  in 
the  field. 


13.  PROTOTYPING  OF  COMPETING  SYSTEMS 

When  a  new  concept  in  ship  design  is  being  considered  for  incorpora¬ 
tion  into  the  fleet  inventory,  the  weapon/sensor/propulsion  system 
suite  which  is' ultimately  chosen  should  be  based  in  large  part  on 
the  DT&E  and  OT&E. 


Extensive  prototype  testing  will  probably  be  required  to  demonstrate 
the  effectiveness  of  various  alternatives.  For  example,  the  necessity  for 
evaluating  different  combat  subsystems  suites  or  variations  in  the  propul¬ 
sion  and  control  subsystems  may  make  it  more  economical  to  construct  more 
than  one  prototype  in  order  to  achieve  timely  test  results  for  the  weapon 
system  milestone  decisions. 

14.  T&E  OF  COMMANP/OPERATING  CENTERS 

The  requirements  of  effective  command  and  control  in  an  operational 
fleet  environment  lead  to  the  need  for  various  information,  support, 
and  command  centers  to  be  provided  aboard  ship. 

Mock-up,  test  and  re-design  iterations  will  normally  be  conducted  in 
the  laboratory  in  the  process  of  ship  system  design.  In  order  to  provide  a 
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maximum  probability  of  success  in  this  endeavor,  it  is  important  to  bring 
OPTEVFOR  into  the  process  at  an  early  date,  and  at  some  point  in  the  project 
have  an  adequate  IOT&E  with  fleet  personnel  manning  the  system.  Depending 
on  the  type  of  personnel/equipment  center  involved,  it  may  be  possible  to 
integrate  this  into  an  existing  ship  for  further  IOT&E  prior  to  settling  on 
a  final  configuration,  and  this  should  be  done  if  at  all  feasible. 


Land  based  test  sites  constructed  for  this  type  of  activity  should 
faithfully  represent  the  equipment  arrangement  anticipated  for  the  new  ship 
class,  including  such  items  as  cabling,  piping,  displays,  communications 
circuits,  and  important  structural  features.  When  applicable,  full  equip¬ 
ment  installations  including  radar  antennas,  directors  and  gun/missile 
launching  and  handling  facilities  should  be  provided.  Location  of  an  AAW 
combat  system  test  site  should  be  such  that  air  targets  can  be  flown 
against  the  system. 
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15.  SOFTWARE  TESTING 

In  view  of  increasing  dependence  upon  computers  for  ship  management 
and  tactical  operation,  software  testing  must  be  exceptionally 
thorough,  and  Integrated  software  testing  must  begin  as  early  as 
possible. 

All  computer-managed  functions  should  be  tested  in  conjunction  with  other 
combat  system  equipment  either  at  land-based  or  at-sea  test  sites.  Data 
inputs  which  are  not  availauie  in  actuality  should  as  a  minimum  be  simulated. 
Testing  should  include  operation  in  back-up  modes,  effects  of  external  elec¬ 
tronic  countermeasures,  effects  of  electromagnetic  pulse  (EMP).  Where  a  system 
is  designed  to  include  secure  data  circuits  within  the  ship,  T&E  should  in¬ 
clude  examination  by  experienced  COMMSEC  technicians  to  discover  possible 
susceptibility  to  compromise.  Extensive  testing  of  the  system  with  whip 
antennas  installed  should  be  conducted  to  determine  nature  and  extent  of 
EMI/RFI.  Testing  is  also  required  to  determine  if  corrective  actions  have 
solved  EMI/RFI  problems. 
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16.  NEW  HULL  FORMS 


When  a  new  type  of  ship  Involves  a  radical  departure  from  the  con¬ 
ventional  hull  form,  extensive  prototype  testing  prior  to  further 
commitment  to  the  new  hull  form  should  be  required. 

This  Is  required  since  the  Froude  Scaling  Laws,  which  traditionally 
allow  proceeding  directly  from  model  testing  to  full-size  hull  with  cer¬ 
tain  knowledge  of  the  latter's  performance,  will  no  longer  fully  apply. 

17.  EFFECT  OF  HULL  AND  PROPULSION  ON  MISSION  CAPABILITIES 

The  predicted  effect  of  the  proven  hull  and  propulsion  system  design 
on  the  performance  of  the  ship's  critical  systems  should  be  deter¬ 
mined  . 

Proven  hull  and  propulsion  system  designs  can  sometimes  turn  out  to 
be  unsatisfactory  when  the  ship  is  to  be  used  for  certain  missions  and  the 
corresponding  critical  systems  to  accomplish  the  missions  are  installed. 
For  example,  a  proven  design  may  have  a  self-noise  configuration  that  is 
totally  unsatisfactory  for  ASW  missions. 

18.  ADVANCES  IN  PROPULSION 

Demonstration  of  the  use  of  new  propulsion  systems  should  be  con¬ 
ducted  prior  to  making  the  decision  to  commit  the  propulsion  system 
to  the  ship  in  question. 

Advances  ip  ship  propulsion  contemplated  for  new  ships  may  take  the 
form  of  scaling  up  systems  or  portions  of  systems  which  have  -previously 
seen  satisfactory  service  in  smaller  size.  An  example  is  the  controll¬ 
able  reversible-pitch  propeller.  When  such  scaling  up  is  contemplated, 
it  is  important  to  conduct  adequate  testing  of  the  larger  model,  in  a 
configuration  as  close  as  possible  to  the  intended  version,  prior  to 
making  an  irrevocable  commitment  of  the  program  to  the  design. 


19.  PROPULSION  SYSTEM  IN  USE  IN  OTHER  CLASSES 

When  an  engine  type  to  be  used  in  the  propulsion  system  of  a  new 
ship  is  already  performing  satisfactorily  in  another  ship,  this 
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is  not  to  be  taken  as  an  indication  that  shortcuts  can  be  taken 
in  propulsion  system  DT&E,  or  that  no  problems  will  be  encountered. 

It  is  possible  that  redesign  will  be  necessary  for  some  components 
of  the  power  transmission  system  (i.e.,  shafting,  gearing,  couplings, 
bearings)  before  a  satisfactory  system  is  developed. 

20.  IOT&E  SHIPBOARD  GUN  SYSTEMS 

Operational  tests  of  shipboard  gun  systems  should  simulate  the 
stress,  exposure  time  and  other  conditions  of  battle  so  that  the 
suitability  of  the  weapon  can  be  evaluated  in  toto. 

Tests  should  include  enough  rounds  fired  under  these  conditions  to 
determine: 

•  Weapon  accuracy  during  ship  maneuvers  and  under  other  stress 
conditions  including  ability  to  maintain  alignment 

•  Smoke  and  toxic  gas  build-up  in  gun  mount  and  adjacent  spaces 
and  other  hazards  to  the  gun  crew 

•  Physical  and  psychological  limitations  of  gun  crew 

•  Inefficiencies  in  ammunition  handling  required  of  personnel 

•  An  estimate  of  barrel  wear  rate,  and  ease  of  replacement 

•  Mean  time  to  failure  of  critical  components 

•  Suitability  of  construction  and  mounting  of  control  equipment 

•  Susceptibility  to  spray  and  water  intrusion 

•  Effects  of  shock,  vibration  and  temperature 

•  Adequacy  of  spare  parts  provided 

•  Adequacy  of  recommended  maintenance  procedures 

•  Adequacy  of  documentation 

21.  TARGETS  FOR  AAW  IOT&E 

Operational  test  of  shipboard  AAW  weapons  demands  the  use  of  targets 
which  realistically  simulate  the  present-day  threat. 

One  of  the  most  likely  targets  is  the  high-speed,  low-altitude  anti¬ 
ship  missile,  and  a  target  that  realistically  simulates  this  threat  may  be 
difficult  to  obtain.  IOT&E  planning  should  provide  in  any  case  for  adequate 
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targets,  and,  if  necessary,  should  provide  for  the  development  of  new  targets 
concurrently  with  that  of  the  system  to  be  tested. 


22.  WAIVERS  TO  T&E  OF  SHIP  SYSTEMS 

Waivers  to  T&E  of  pre-production  models  of  a  system  in  order  to  speed 
up  production  and  delivery  should  be  made  only  after  consideration  of 
all  costs  and  benefits  of  the  waiver,  including  those  not  associated 
with  the  contract. 


A  waiver  to  speed  up  delivery  may  simply  shift  the  additional  cost 
of  back-fitting  or  modification  of  defects  that  could  have  been  detected 
by  early  T&E  to  another  contract  or  budget  classification.  Back-fitting 
via  Engineering  Change  Proposals  is  extremely  expensive.  In  terms  of 
operational  time  the  net  effect  of  the  production  speed-up  may  be  minimal. 


23.  ENVIRONMENTAL  EFFECTS  ON  SONAR  DOMES 

Environmental  effects  on  sonar  domes  and  their  self-noise  should 
be  tested  and  evaluated  before  the  domes  are  accepted  as  part 
of  the  sonar  system  concept. 


Excessive  pitting  and  salt  water  corrosion  can  cause  degradation  in 
sound  transmission  and  increases  in  self  noise  which  would  require  frequent 
off-r.tation  maintenance. 


24.  HULL/MACHINERY  TESTING  BY  COMPUTER  SIMULATION 

In  DT&E  of  ships,  there  will  be  cases  where  the  best  means  to 
conduct  evaluations  of  particular  hull  and  machinery  capabilities 
is  through  dynamic  analysis  using  computer  simulation,  with  later 
validation  of  the  simulation  by  actual  test. 

An  example  is  shock  hardness  testing.  Experience  has  shown  that  the 
concept  is  not  understood  by  all  vendors,  and  careful  selection  will  be 
necessary  to  assure  competence.  If  further  training  of  a  selected  con¬ 
tractor  in  the  use  of  this  tool  will  be  necessary,  the  test  plan  should 
so  provide. 
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25.  OPERATIONAL  RELIABILITY 

IOT&E  should  provide  valuable  data  on  the  operational  reliability 
of  ship  weapon  systems  which  cannot  be  obtained  through  DT&E. 


Factors  such  as  operator  error  failures,  and  environmentally  induced 
failures  should  be  looked  for  in  the  operational  tests  and  investigated  to 
determine  if  system  deficiencies  are  underlying  reasons  for  the  failures. 
Especially  important  is  the  procedure  used  to  evaluate  the  operational 
reliability  of  the  system  as  determined  by  the  relatively  small  amount  of, 
but  significant,  data  obtained  through  IOT&E  and  the  larger  amounts  of  data 
on  hardware  design  reliability  collected  through  DT&E.  Further,  the  main¬ 
tenance  practices  should  be  carefully  studied  to  assess  its  impact  on  the 
observed  operational  reliability  obtained  through  IOT&E.  It  may  be  desir¬ 
able  to  require  contractor  technical  personnel  during  DT&E  to  use  exact 
maintenance  procedures  contained  in  technical  manuals  to  ensure  that  these 
procedures,  diagrams,  and  wiring  charts  are  correct  and  satisfactory. 

26.  INTER- SHIPYARD  COORDINATION 

In  general,  ships  are  built  and  tested  by  more  thin  one  shipyard 
or  company.  All  plans  for  T&E  should  be  analyzed  to  ensure  that 
inter-shipyard  coordination/cooperation  is  possible. 

Testing  procedures  designed  by  shipbuilders  do  not  generally  draw 
upon  experience  of  other  yards.  Where  it  is  possible  to  draw  on  these 
experiences,  the  contract,  organization,  legal  and  institutional  frame¬ 
work  of  the  procurement  should  encourage  such  coordination/cooperation. 

For  example,  the  designer-builder-supplier  T&E  coordination  plan  should 
be  requested;  the  T&E  plan  should  be  applicable  to  all  builders;  the 
constraints  on  interaction  of  builders  should  be  carefully  examined. 
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III.  FULL-SCALE  ENGINEERING  DEVELOPMENT  PHASE 


Ideally,  contracts  for  detailed  design  and  later  construction  of  a 
full-scale  prototype  or  lead  ship  will  be  awarded  during  this  phase. 

During  full-scale  development,  remaining  DT&E,  and  IOT&E  of  individual 
combat  subsystems  will  be  completed.  Additional  phases  of  IOT&E  may  be 
performed  with  pilot  or  pre-production  items  of  the  individual  subsystems 
as  necessary  to  gain  approval  for  service  use,  if  possible,  prior  to  T&E 
of  the  integration  of  the  combat  system.  Integration  testing  will  normally 
take  place  at  a  land-based  or  sea-based  test  installation  built  as  nearly 
as  possible  to  the  same  configuration  of  the  intended  ship  installation. 

It  should  include  software  integration,  as  well  as  hardware  integration 
testing.  (In  addition,  there  may  be  afloat  IOT&E  in  a  prototype  ship, 
although  this  will  not  normally  be  the  case.)  Sufficient  test  and  eval¬ 
uation  is  conducted  to  reduce  risk  to  the  point  that  a  decision  regarding 
quantity  production  can  be  made. 

Checklist  items  for  this  phase  are  as  follows: 

1.  Initial  or  Pilot  Phase  of  IOT&E 

2.  Value  of  Operational  Testing 

3.  Personnel  Performance  in  Concurrent  Evaluations 

4.  Identify  Critical  Subsystems 

5.  Reliability  of  Critical  Systems 

6.  Consistency  in  Test  Objectives 

7.  Single  Screw  Ship 

8.  Problems  Associated  with  New  Hulls 

9.  Effect  on  Personnel  of  Hull/Propulsion  Innovations 

10.  Controlled  Environment 

11.  Reliability/Maintainability:  Personnel  Interactions 

The  reader  should  also  refer  back  to  the  checklist  items  in  the  pre¬ 
vious  phases  since  many  of  these  items  will  be  applicable  to  this  phase. 


Preceding  page  blank 
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1.  INITIAL  OR  PILOT  PHASE  OF  IOT&E 

Before  any  operational  tests  for  demonstration  of  operational  suit¬ 
ability  and  effectiveness  are  conducted,  an  Initial  or  pilot  test 
should  be  conducted. 

The  primary  purpose  of  the  test  is  shaking  down  the  test  plan,  and 
briefing  or  training  participants  as  necessary  regarding  the  instrumenta¬ 
tion  concept,  the  data  analysis  plan,  and  other  test  features.  This  phase 
should  be  conducted  early  enough  so  that  sufficient  time  is  available  to 
make  the  necessary  changes  to  the  IOT&E  plan  as  dictated  by  the  results 
of  the  pilot  test.  In  the  case  of  ship  weapon  system  testing,  the  scarcity 
of  test  range  time  and  test  services  dictate  the  need  to  conduct  such  a 
pilot  phase  to  avoid  waste  of  these  valuable  services. 


2.  VALUE  OF  OPERATIONAL  TESTING 

Operational  tests  should  be  well  planned  and  scheduled. 


Operational  testing  is  essential,  but  it  is  also  expensive  and  time 
consuming.  To  ensure  efficient  use  of  submarine,  aircraft,  ship  or  range 
test  support  units,  test  concepts  should: 

(a)  Involve  organizations  with  operational  experience  in  estab¬ 
lishing  measures  of  effectiveness,  so  that  the  outcome  of 
the  tests  will  be  accepted  as  being  both, technically  and 
operationally  significant. 

(b)  Determine  whether  the  scope  of  the  planned  tests  will  pro¬ 
vide  sufficient  data  to  justify  any  change  at  all  in  the 
eyes  of  potential  users. 

(c)  Compare  the  scope  of  proposed  tests  against  checklists  of 
issues  frequently  raised  at  major  decision  milestones, 
including  those  contained  herein,  to  assure  that  the  data 
needed  for  such  decisions  will  be  forthcoming  to  the 
extent  this  is  possible  from  testing  alone. 

Tests  should: 

(a)  Have  specific  objectives; 

(b)  Be  instrumented  to  permit  diagnosis  of  the  causes  of  lack 
of  performance  including 
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•  "Random"  failures 

•  Design-induced  failures 

•  Wear  out  failures 

•  Operator  error  failures 

•  And  those  as  a  result  of  accidental  environmental  con¬ 
ditions; 

(c)  Be  scheduled  to  permit  analyses  of  the  methods  available  to 
cure  each  class  of  failures  to  corrective  actions  can  be 
taken; 

(d)  Never  be  repeated  if  failures  occurred  without  a  detailed 
analysis  of  the  failure.  Most  likely,  the  failure  will  not 
go  away. 

3.  PERSONNEL  PERFORMANCE  IN  CONCURRENT  EVALUATIONS 

Evaluation  of  test  results  must  take  into  account  the  expertise 
achieved  by  T&E  personnel  through  long  association  with  the  T&E 
program. 

Weapon  systems  are  often  put  through  the  process  of  a  concurrent 
evalution  through  the  use  of  sequential  DT&E  and  IOT&E.  Often,  the  same 
operating  and  maintenance  personnel  are  present  and  participate  in  equip¬ 
ment  installation  and  checkout,  DT&E  and  finally  IOT&E.  In  these  cases, 
and  particularly  if  the  test  program  has  been  long  and  difficult,  a  re¬ 
sult  is  that  IOT&E  is  conducted  with  operating  and  maintenance  personnel 
whose  skills  have  been  peaked  through  long  association  with  the  new  equip¬ 
ment.  Since  they  may  also  have  been  initially  selected  for  their  aptitude, 
they  can  in  no  sense  be  considered  typical.  In  particular,  new  ships 
normally  not  only  are  manned  with  100  percent  of  allowance  by  rate,  but 
also  receive  special  pre-commissioning  training  on  the  subsystems  incor¬ 
porated  in  the  new  ship  not  representative  of  the  average  routine  person¬ 
nel  replacement.  This  and  other  bias  in  the  data  which  supports  the 
overall  results  and  evaluation  thereof  must  be  eliminated  if  possible, 
but  if  not  eliminated,  then  recognized  and  highlighted  in  the  evaluation 
of  results. 
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4. 


IDENTIFY  CRITICAL  SUBSYSTEMS 


In  the  planning  for  the  IOT&E  of  a  ship  system,  the  critical  sub¬ 
systems  with  respect  to  mission  performance  should  be  Identified. 

Adequate  measures  of  effectiveness  of  these  subsystems  must  be  incor- 
proated  into  the  ship  IOT&E.  The  schedule  of  production  and  testing  should 
be  carefully  examined  to  ensure  that  time  is  available  for  sufficient  and 
necessary  subsystem  and  system  testing  and  that  all  T&E  activities,  which 
are  many,  are  coordinated.  Fallback  positions  should  be  addressed  in  the 
planning  process. 

5.  RELIABILITY  OF  CRITICAL  SYSTEMS 

T&E  should  determine  the  expected  reliability  at  sea  of  systems 
critical  to  the  ship's  mobility  and  primary  and  major  secondary 
tasks. 

Frequent  unscheduled  maintenance  in  port  severely  limits  the  opera¬ 
tional  readiness  of  a  ship,  and  ultimately  of  its  parent  unit.  Examples 
of  critical  systems  where  crippling  reliability  problems  have  occurred 
are: 

•  High  pressure  boilers 

•  Sonar  domes 

•  Fropeller  shafts  and  journal  bearings 

•  Steering  mechanisms 

•  Anchor  mechanisms 

•  Towed  body  hoists  and  towed  body  assemblies 

•  Reduction  gears 

•  Thrust  bearings 

•  Evaporators 

•  Oxygen  generators 

6.  CONSISTENCY  IN  TEST  OBJECTIVES 

There  are  various  phases  of  testing  of  a  ship  system.  One  should 
ensure  that  the  objectives  of  one  phase  are  not  inconsistent  with 
the  objectives  of  the  other  phases. 
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If  inconsistencies  occur,  builders  and  testers  could  become  con¬ 
fused,  and  test  redundancies,  delays  and  ultimately  higher  program  costs 
will  result.  It  is  highly  desirable  to  review  all  the  T&E  procedures  and 
acceptance  standards  prior  to  the  acquisition  decision  to  ensure  con¬ 
sistency. 

7.  SINGLE  SCREW  SHIPS 

T&E  of  the  propulsion  systems  of  ships  with  a  single  screw  should 
be  especially  rigorous  to  determine  failure  rates,  maintenance  and 
repair  alternatives. 

A  failure  in  the  propulsion  system  of  a  single  screw  ship  is 
especially  critical,  because  the  ship  has  no  auxiliary  system  to  bring 
it  to  port  under  its  own  power. 

8.  PROBLEMS  ASSOCIATED  WITH  NEW  HULLS 

Whenever  a  new  hull  is  incorporated  into  the  ship  design,  a  test 
and  evaluation  of  this  hull  should  be  done  prior  to  the  full-scale 
production  and  incorporation  of  the  major  weapons  subsystems. 

For  example,  tests  should  be  conducted  to  ensure  that  prqblems  asso¬ 
ciated  with  a  hull  design  which  takes  on  board  a  significant  amount  of 
water  will  not  significantly  affect  the  utility  of  various  subsystems  on 
the  ship. 

9.  EFFECT  ON  PERSONNEL  OF  HULL/PROPULSION  INNOVATIONS 

Innovations  in  hull  design  and  propulsion  systems  can  be  expected 
to  cause  human  factor/habitability  problems  easily  overlooked  in  the 
search  for  technical  excellence. 

These  factors  should  be  evaluated  by  realistic  test  in  actual  opera¬ 
tions  and  results  given  equal  weight  with  other  performance  factors.  In 
making  the  evaluations,  it  must  be  realized  that  the  degree  of  comfort 
required  to  live  in  a  ship  and  operate  it  effectively  exceeds  that  required 
to  survive  aboard  for  short  periods.  Thus,  a  ship  with  a  continuous  high 
noise  level  from  a  new  type  propulsion  system,  or  one  with  a  peculiar 
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motion  that  induces  seasickness  in  a  significant  proportion  of  the  crew 
would  be  unacceptable  as  a  combat  unit,  even  though  technical  and  opera¬ 
tional  performance  features  rendered  it  otherwise  suitable. 

10.  CONTROLLED  ENVIRONMENT 

It  is  important  that  the  installation  of  the  system  for  IOT&E  be  in 
an  environment  controlled  with  no  greater  stringency  than  the 
intended  shipboard  installation. 

Modern  shipboard  weapon  and  command  and  control  systems  usually  re¬ 
quire  a  controlled  environment,  i.e.,  dry  air  and  cooling,  for  proper 
and  reliable  operation. 

11.  RELIABILITY/MAINTAINABILITY:  PERSONNEL  INTERACTIONS 

Reliability  and  maintainability  assessments  of  equipment  for  instal¬ 
lation  aboard  ships  should  be  designed  with  due  regard  for  the 
personality  and  motivations  of  the  crew  members  who  will  operate  and 
maintain  the  equipment  in  actual  use. 

The  man  should  be  considered  part  of  the  system  and  his  interaction 
with  other  parts  of  the  system  should  be  subjected  to  IOT&E.  For  example, 
if  an  equipment  depends  for  its  water  tightness  on  a  metal  cover  secured 
by  twenty  screws,  and  IOT&E  reveals  a  congenital  unwillingness  on  the 
part  of  the  enlisted  technician  to  repeatedly  secure  them,  then  the  equip¬ 
ment  will  have  to  be  redesigned  since  the  man  cannot  be.  It  is  axiomatic 
that  essential  maintenance  operations  which  must  be  frequently  performed, 
and  which  at  the  same  time  are  difficult,  awkward  or  arduous,  will  be 
short-circuited  or  bypassed  in  some  manner  by  the  technician.  Widespread 
failures  of  shipboard  equipment  which  are  reportedly  the  result  of  lack  of 
routine  care,  can  often  be  traced  to  insufficient  regard  for  this  princi¬ 
ple  in  early  testing. 
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IV.  SUBSTANTIAL  PRODUCTION /DEPLOYMENT  PHASE 


The  period  after  quantity  procurement  of  ships  of  the  new  class 
commences  is  the  Production/Deployment  Phase.  It  is  probable  that  delivery 
of  the  lead  ship  of  the  class  will  occur  during  this  phase,  and  acceptance 
trials  and,  if  required,  whole-ship  OT&E  will  take  place.  Results  of  T&E 
of  the  lead  ship  as  it  is  put  through  its  paces  will  result  in  important 
changes  which  can  be  fed  back  into  follow  ships.  Subsystem  OT&E  which  re¬ 
quires  the  actual  ship  must  also  await  this  phase. 

Checklist  items  are  as  follows: 

1.  Design  of  Ship  FOT&E 

2.  Operational  Testing  During  Shakedown  Period 

3.  Fleet  Operations  in  FOT&E 

4.  Ship  ASW  OT&E  Planning 

5.  Variable  Depth  Sonar  OT&E 

6.  Ship  Self-Noise  Tests 

7.  Effect  of  Major  ECM  on  Ship  Capability 

8.  Ship  System  Survivability 

9.  Interlocks 

10.  Intra-Ship  Communications 

The  reader  should  review  the  previous  checklist  items  since  many  of 
them  may  be  applicable  to  this  phase. 
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1.  DESIGN  OF  SHIP  FOT&E 

In  the  testing  program  of  a  ship  system,  it  should  be  recognized 
that  although  it  may  be  designated  as  a  special  purpose  ship.  It 
will  in  most  cases  be  used  In  a  general  purpose  role  as  well. 


Thus,  the  T&E  program  should  address  the  utility  of  the  ship  system 
in  the  other  roles  within  its  intended  general  capability.  In  addition 
to  special  mission-related  T&E,  OT&E  to  be  conducted  on  the  lead  ship  of 
a  class  should  be  designed  to  show  its  ability  to  perform  the  routine 
evolutions  that  all  ships  must  do  regularly  during  normal  fleet  opera¬ 
tions.  The  following  are  examples  of  recommended  objectives  in  this  period: 

(a)  Investigate  and  measure  replenishment  capabilities 
of  the  ship,  including  strikedown  of  stores.  For 
helicopter  replenishment,  investigate  the  effects 
of  helicopter  service  and  maintenance  requirements 
on  movement  and  strikedown  of  stores. 

(b)  Examine  weapons  handling,  assembly,  loading,  and  dis¬ 
posal  under  at-sea  conditions,  and  adequacy  of  magazine 
stores  to  a  tactical  situation.  On  aircraft  carriers, 
test  to  determine  whether  safe  weapon  stowage  and  handling 
practices,  including  radiation  hazards,  can  be  main¬ 
tained  under  conditions  of  sortie  generation  at  rates 
expected  in  tactical  operations. 

(c)  If  helicopters  are  to  be  operated  from  the  ship,  test/ 
examine  limitations  on  courses  and  speeds  in  helicopter 
operations,  resulting  from  superstructure  or  stack 
arrangements.  Relate  these  to  tactical  situations  and 
requirements. 

(d)  For  escort  vehicles,  including  SSN  in  the  escort  role, 
test  to  ensure  that  the  ship  can  operate  effectively 
with  escorted  merchant  shipping  in  terms  of  communica¬ 
tions  and  data  transfer.  Test  susceptibility  to  counter¬ 
measures. 

(e)  Test/examine  the  repairability/maintainability  of  essential 
equipment  in  terms  of  accessibility,  fault  location,  and 
fault  diagnosis  as  installed  in  the  ship.  Report  on 
adequacy  of  manuals  and  diagrams  for  repair  and  maintenance 
under  these  conditions. 

(f)  Recognizing  that  realistic  testing  in  this  regard  is  diffi¬ 
cult  to  schedule  because  of  shortage  of  services  and  expense 
of  artificial  testing,  every  effort  should  be  made  by  T&E 
planners  to  incorporate  the  lead  ship  as  soon  as  possible 
into  active  fleet  operations,  with  experienced  observers 
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reporting.  Fleet  exercises  present  the  most  varied  opportuni¬ 
ties  possible  short  of  actual  deployment,  and  consideration 
should  be  given  to  participation  by  the  lead  'ship  in  these 
operations  for  T&E  purposes. 


2.  OPERATIONAL  TESTING  DURING  SHAKEDOWN  PERIOD 

The  time  period  for  OT&E  of  a  ship  can  be  used  more  efficiently  if 
full  advantage  is  taken  of  periods  immediately  after  the  ship  is 
delivered  to  the  Navy,  i.e.,  during  the  shakedown  period,  final  con¬ 
tract  trials,  and  immediately  after  PSA. 


Consideration  should  be  given  to  extending  the  shakedown  periods  for 
the  first  ship  from  each  building  yard  of  multiple-yard  contracts  so  that 
vital  operational  tests  can  be  conducted.  In  addition,  standard  tests 
(e.g.,  WSAT,  SQT,  COT)  might  be  expanded  in  order  to  cover  any  test 
criteria  not  otherwise  included  in  the  program.  Joint  planning  to  in¬ 
clude  COMOPTEVFOR  is  necessary  to  accomplish  this  effectively. 


3.  FLEET  OPERATIONS  IN  FOT&E 

A  great  deal  of  information  on  the  operational  effectiveness  of  a 
ship  can  be  obtained  from  standard  fleet  operations  through  well 
designed  information  collection,  processing,  and  analysis  procedures. 

Such  procedures  should  be  regarded  as  a  necessary  adjunct  to  OT&E 
and  should  be  utilized  by  COMOPTEVFOR  in  assessing  the  performance  of  a 
new  ship  class,  i.e.,  this  kind  of  data  will  provide  a  valuable  baseline 
from  which  to  evaluate  new  ship  classes.  The  3M  system.  Fleet  ASW  Data 
Acquisition  Program  (FADAP) ,  and  reports  of  fleet  operational  exercises 
are  examples  of  this  kind  of  data. 


4.  SHIP  ASW  OT&E  PLANNING 

In  planning  OT&E  of  shipboard  systems,  it  is  important  to  recognize 
the  difficulty  of  achieving  realism,  perhaps  more  so  than  in  other 
areas  of  naval  warfare. 


This  is  because  fleet  operations  involve  many  hours  of  no  contact 


which  may  lead  to  operator  boredom  and  inattention,  and  many  false 
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contacts  for  every  real  contact.  The  need  for  economical  utilization 
of  curtailed  operating  time  and  scarce  target  services  thus  militates 
directly  against  realistic  testing. 

5.  VARIABLE  DEPTH  SONAR  OT&E 

The  behavior  of  towed  bodies  of  variable  depth  sonar  systems  and 
towed  arrays  should  be  tested  and  evaluated  under  all  ship  maneuvers 
and  speeds  likely  to  be  encountered  in  combat. 

This  should  include  tests  of : 

•  Kiting  effect 

•  Retrieving  the  towed  body 

•  Gyro  stability 

•  Heading  stabilization  times  after  maneuvers 

•  Streaming  depth 

•  Bearing  and  distance  accuracy  while  under  maneuvers 

•  Self  noise 

6.  SHIP  SELF-NOISE  TESTS 

The  magnetic  and  acoustic  signatures  of  a  ship  can  be  tested  accu¬ 
rately  only  after  it  is  completed. 

Furthermore,  own  ship  machinery  noise  spikes  on  own  sonar  can  be 
observed,  analyzed,  and/or  corrected  at  this  time.  The  settings  and 
power  required  for  effective  operation  of  the  degaussing  system  can  be 
determined  at  this  time.  Therefore,  these  types  of  testing  should  be 
performed  as  soon  as  possible  after  the  lead  ship  is  completed  so  as  to 
permit  feedback  into  follow-on  ships. 

7.  EFFECT  OF  MAJOR  ECM  ON  SHIP  CAPABILITY 

The  FOT&E  of  a  ship  should  include  tests  of  the  effectiveness  of 
the  ship  when  subjected  to  major  ECM. 

This  normally  cannot  be  done  at  land-based  test  sites:  therefore  the 
tests  should  be  performed  on  the  first  ship  of  the  class.  Tests  should 
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not  be  confined  to  the  effect  on  equipment,  but  should  consider  human 
factors  (stress,  mental  confusion,  COC  problems,  etc.). 

8.  SHIP  SYSTEM  SURVIVABILITY 

Operational  test  and  evaluation  of  modern  ships  should  provide  for 
the  assessment  of  their  ability  to  survive  and  continue  to  fight 
when  subjected  to  battle  damage. 

This  is  particularly  important  today  because  so  much  of  the  ship's 
combatant  capability  depends  on  the  functioning  of  a  complex  computer 
network,  and  because  of  the  lack  of  semi-automatic  and  manual  modes  of 
operation  in  some  modem  weapons.  Results  of  such  testing  in  a  land 
based  test  site  will  serve  to  highlight  the  need  for  additional  flexi¬ 
bility  or  degraded  modes  of  operation  in  the  command  and  control  or  weapon 
system.  When  conducted  in  a  prototype  or  lead  ship,  this  kind  of  testing 
may  show  that  additional  redundancy  in  equipment  or  cabling  is  required 
in  the  follow  ships. 

9.  INTERLOCKS 

Shipboard  electronic  systems  are  designed  with  interlock  switches 
that  open  electrical  circuits  for  safety  reasons  when  equipment 
cabinets  are  opened.  OT&E  should  be  able  to  detect  over-design  as 
well  as  minimum  design  adequacy  of  interlock  systems. 

As  an  example  of  over-design,  one  gun  fire  control  system  had  inter¬ 
locks  which  deactivated  unnecessarily  the  anti-aircraft  fire  control  por¬ 
tion  of  the  system  when  the  surface  fire  control  cabinets  were  opened  for 
maintenance,  and  vice  versa.  Interlock  switches  should  be  in  a  pyramid, 
with  only  a  few  major  ones  that  affect  the  entire  system.  Where  equip¬ 
ments  feed  computer  input  units,  dual  interlocks  should  be  provided  that 
first  open  the  data  lines  and  second  shut  down  equipment  power.  Good  OT&E 
can  provide  timely  recommendations  for  equipment  modification. 

10.  INTRA-SHIP  COMMUNICATIONS 

In  the  conduct  of  lead  ship  trials  and  evaluations,  particular 
attention  should  be  given  to  the  operational  impact  resulting 


from  the  absence,  by  design,  of  intra-ship  communications  circuits 
and  stations  from  important  operating  locations. 

As  the  most  reliable  form  of  intra-ship  communications,  the  sound- 
powered  circuits  should  be  given  especially  critical  examination.  Often 
the  addition  of  a  new  circuit,  or  a  new  station  on  an  existing  circuit 
will  be  the  most  effective  means  of  remedying  an  operational  problem  that 
doesn't  become  evident  until  the  lead  ship  goes  to  sea,  and  one  that  can 
be  economically  incorporated  in  follow  ships. 
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